Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](metrics) Unify metrics of thread pool #43144

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

[feat](metrics) Unify metrics of thread pool #43144

wants to merge 7 commits into from
+420 −170

Conversation

zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Nov 2, 2024 •
edited
Loading

What problem does this PR solve?

Add metrics for all thread pool, more specifically, for all ThreadPool objects.
All thread pool will have following metrics:

  1. thread_pool_active_threads
  2. thread_pool_queue_size
  3. thread_pool_max_queue_size
  4. thread_pool_max_threads
  5. task_execution_time_ns_avg_in_last_1000_times
  6. task_wait_worker_ns_avg_in_last_1000_times

A new class IntervalHistogramStat is created for interval histogram calculation.

Metrics is updated by hook method when they are needed by prometheus.

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Nov 2, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exec/scan/scanner_scheduler.cpp Outdated Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9821/25967)
Line Coverage: 28.99% (81627/281564)
Region Coverage: 28.25% (42144/149158)
Branch Coverage: 24.83% (21381/86106)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b36415796af4881788f26f82f81791a16e8e4608_b36415796af4881788f26f82f81791a16e8e4608/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as draft November 4, 2024 00:59
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

1 similar comment
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9820/25967)
Line Coverage: 28.99% (81616/281564)
Region Coverage: 28.25% (42141/149158)
Branch Coverage: 24.83% (21379/86106)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b36415796af4881788f26f82f81791a16e8e4608_b36415796af4881788f26f82f81791a16e8e4608/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9821/25970)
Line Coverage: 28.97% (81596/281672)
Region Coverage: 28.24% (42131/149203)
Branch Coverage: 24.82% (21381/86140)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7eb8642f9a587895a5c757f61d356507495daed1_7eb8642f9a587895a5c757f61d356507495daed1/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

image

Fail of test has nothing to do with this pr.

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review November 4, 2024 12:51
wangbo
wangbo previously approved these changes Nov 6, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by anyone and no changes requested.

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as draft November 6, 2024 09:32
github-actions[bot]
github-actions bot reviewed Dec 17, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/util/interval_histogram.h Show resolved Hide resolved
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh changed the title [opt](metrics) More metrics for scanner [feat](metrics) Unify metrics of thread pool Dec 17, 2024
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review December 19, 2024 12:29
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

1 similar comment
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39916 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3d0859e06809be66920e7651a4fe6b505e50f751, data reload: false

------ Round 1 ----------------------------------
q1	17590	7413	7282	7282
q2	2051	185	168	168
q3	10578	1116	1138	1116
q4	10478	733	770	733
q5	7616	2712	2729	2712
q6	241	153	148	148
q7	1020	647	599	599
q8	9255	1899	1924	1899
q9	6699	6487	6470	6470
q10	7039	2375	2323	2323
q11	470	269	261	261
q12	437	223	218	218
q13	17764	2977	2911	2911
q14	240	220	206	206
q15	552	500	501	500
q16	648	587	585	585
q17	1001	557	618	557
q18	7335	6587	6812	6587
q19	1348	960	947	947
q20	489	184	192	184
q21	4034	3273	3206	3206
q22	373	315	304	304
Total cold run time: 107258 ms
Total hot run time: 39916 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7272	7248	7232	7232
q2	330	238	232	232
q3	2903	2828	2978	2828
q4	2053	1832	1816	1816
q5	5677	5688	5630	5630
q6	238	144	148	144
q7	2280	1798	1841	1798
q8	3392	3538	3509	3509
q9	8973	8967	9026	8967
q10	3627	3608	3552	3552
q11	610	520	509	509
q12	826	622	605	605
q13	12033	3095	3095	3095
q14	311	274	286	274
q15	570	537	516	516
q16	693	645	658	645
q17	1891	1655	1624	1624
q18	8356	7899	7834	7834
q19	1778	1614	1605	1605
q20	2094	1879	1920	1879
q21	5677	5416	5416	5416
q22	643	592	579	579
Total cold run time: 72227 ms
Total hot run time: 60289 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198366 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3d0859e06809be66920e7651a4fe6b505e50f751, data reload: false

query1	1292	935	981	935
query2	6240	2366	2449	2366
query3	11116	4851	4898	4851
query4	32721	23481	23697	23481
query5	4016	470	463	463
query6	284	201	218	201
query7	3983	311	308	308
query8	293	251	236	236
query9	9369	2753	2755	2753
query10	471	238	249	238
query11	17998	15574	15381	15381
query12	158	110	107	107
query13	1578	426	422	422
query14	9796	7302	8014	7302
query15	282	197	197	197
query16	8341	514	517	514
query17	1791	638	660	638
query18	2170	327	338	327
query19	376	172	168	168
query20	127	116	116	116
query21	215	112	114	112
query22	4810	4671	4478	4478
query23	35023	33812	33555	33555
query24	10569	2564	2472	2472
query25	621	395	404	395
query26	1216	165	153	153
query27	2746	346	337	337
query28	7852	2524	2505	2505
query29	847	425	420	420
query30	232	151	153	151
query31	1037	842	853	842
query32	104	65	54	54
query33	745	332	305	305
query34	983	530	522	522
query35	929	772	786	772
query36	1149	954	960	954
query37	133	70	73	70
query38	4344	4179	4337	4179
query39	1547	1545	1478	1478
query40	212	110	101	101
query41	45	46	44	44
query42	122	105	103	103
query43	554	509	508	508
query44	1291	851	818	818
query45	193	168	174	168
query46	1192	734	720	720
query47	2068	1925	1985	1925
query48	441	331	339	331
query49	1003	384	387	384
query50	844	400	394	394
query51	7350	7206	7348	7206
query52	105	91	97	91
query53	263	183	187	183
query54	1182	413	431	413
query55	79	77	81	77
query56	270	265	252	252
query57	1311	1181	1200	1181
query58	244	217	220	217
query59	3439	3211	3160	3160
query60	286	250	257	250
query61	144	107	125	107
query62	912	745	759	745
query63	225	190	193	190
query64	3944	717	663	663
query65	3318	3267	3311	3267
query66	781	309	321	309
query67	16397	15611	15553	15553
query68	5087	555	573	555
query69	475	256	263	256
query70	1231	1126	1143	1126
query71	464	247	260	247
query72	6448	4150	4176	4150
query73	780	374	363	363
query74	9956	8867	8976	8867
query75	3472	2721	2710	2710
query76	3741	1167	1081	1081
query77	666	347	300	300
query78	10479	9549	9686	9549
query79	2005	626	620	620
query80	1078	428	440	428
query81	518	244	229	229
query82	642	120	126	120
query83	205	147	143	143
query84	284	74	135	74
query85	1428	320	314	314
query86	439	307	297	297
query87	4426	4531	4310	4310
query88	4195	2233	2227	2227
query89	432	291	292	291
query90	2068	191	189	189
query91	138	104	104	104
query92	66	50	54	50
query93	2082	553	546	546
query94	867	293	256	256
query95	354	258	252	252
query96	644	277	284	277
query97	2884	2692	2706	2692
query98	219	195	200	195
query99	1727	1470	1430	1430
Total cold run time: 305219 ms
Total hot run time: 198366 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3d0859e06809be66920e7651a4fe6b505e50f751, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.03
query3	0.23	0.07	0.06
query4	1.61	0.11	0.10
query5	0.43	0.42	0.39
query6	1.15	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.51
query10	0.55	0.57	0.57
query11	0.16	0.11	0.10
query12	0.13	0.11	0.12
query13	0.61	0.61	0.59
query14	2.72	2.76	2.82
query15	0.91	0.82	0.82
query16	0.37	0.39	0.39
query17	0.99	1.05	1.03
query18	0.24	0.22	0.20
query19	1.97	1.91	2.02
query20	0.01	0.01	0.01
query21	15.36	0.59	0.59
query22	2.70	2.11	1.84
query23	17.02	1.02	0.90
query24	3.28	2.06	0.79
query25	0.21	0.16	0.08
query26	0.58	0.14	0.14
query27	0.04	0.04	0.04
query28	9.96	1.10	1.06
query29	12.61	3.18	3.20
query30	0.24	0.06	0.06
query31	2.85	0.40	0.38
query32	3.25	0.47	0.46
query33	3.16	3.08	3.11
query34	17.23	4.48	4.52
query35	4.51	4.52	4.58
query36	0.68	0.48	0.47
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.01 s
Total hot run time: 32.67 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

yiguolei
yiguolei reviewed Dec 25, 2024
View reviewed changes
be/src/util/threadpool.h Outdated
@@ -99,7 +104,7 @@ class Runnable {
//
class ThreadPoolBuilder {
public:
explicit ThreadPoolBuilder(std::string name);
explicit ThreadPoolBuilder(std::string name, std::string workload_group = "system");
Copy link
Contributor

@yiguolei yiguolei Dec 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

叫system 可能会有问题，比如真的有一个workload group 叫system ，此时看到的信息是乱的。
如果我们就不设置workload group ，此时metric 会乱吗？

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不会乱

@doris-robot
Copy link

TPC-H: Total hot run time: 33171 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7002be21e84b6cb7d7b04c31d44180b833185210, data reload: false

------ Round 1 ----------------------------------
q1	17585	6280	6022	6022
q2	2050	342	186	186
q3	10362	1259	781	781
q4	10237	897	468	468
q5	7543	2225	2048	2048
q6	222	188	152	152
q7	928	776	617	617
q8	9238	1440	1228	1228
q9	5323	5021	5023	5021
q10	6793	2334	1874	1874
q11	509	290	257	257
q12	363	365	247	247
q13	17774	3577	3035	3035
q14	230	237	218	218
q15	570	513	497	497
q16	655	621	586	586
q17	598	881	333	333
q18	7185	6699	6422	6422
q19	2165	982	599	599
q20	315	339	193	193
q21	3068	2358	2065	2065
q22	377	353	322	322
Total cold run time: 104090 ms
Total hot run time: 33171 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6390	6244	6287	6244
q2	239	343	232	232
q3	2271	2695	2339	2339
q4	1438	1855	1372	1372
q5	4391	4816	4984	4816
q6	190	178	143	143
q7	2127	2067	1839	1839
q8	2706	2827	2757	2757
q9	7431	7321	7307	7307
q10	3078	3431	2751	2751
q11	591	515	483	483
q12	704	791	609	609
q13	3397	3823	3200	3200
q14	297	345	284	284
q15	575	522	513	513
q16	671	688	647	647
q17	1256	1786	1252	1252
q18	7660	7648	7335	7335
q19	913	1101	1230	1101
q20	2073	2037	1900	1900
q21	5792	5414	4913	4913
q22	633	694	631	631
Total cold run time: 54823 ms
Total hot run time: 52668 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198296 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7002be21e84b6cb7d7b04c31d44180b833185210, data reload: false

query1	1312	966	927	927
query2	6482	2321	2471	2321
query3	10965	4676	4682	4676
query4	32826	24004	23673	23673
query5	3748	619	468	468
query6	275	198	190	190
query7	3985	500	314	314
query8	302	249	240	240
query9	9461	2743	2734	2734
query10	431	298	254	254
query11	18031	15327	15496	15327
query12	158	106	106	106
query13	1569	539	415	415
query14	9253	8028	7557	7557
query15	251	208	190	190
query16	8145	588	422	422
query17	1547	773	591	591
query18	2134	437	330	330
query19	216	195	185	185
query20	131	121	125	121
query21	214	137	119	119
query22	4848	4639	4694	4639
query23	34593	33696	33591	33591
query24	6493	2320	2347	2320
query25	519	502	434	434
query26	791	284	159	159
query27	2241	477	353	353
query28	5411	2542	2491	2491
query29	629	568	429	429
query30	212	183	148	148
query31	1015	907	864	864
query32	78	58	55	55
query33	501	354	303	303
query34	786	891	536	536
query35	810	852	770	770
query36	1020	1057	960	960
query37	120	104	75	75
query38	4452	4358	4392	4358
query39	1562	1480	1460	1460
query40	211	115	106	106
query41	46	44	42	42
query42	132	107	109	107
query43	538	552	497	497
query44	1386	826	834	826
query45	185	183	176	176
query46	911	1091	686	686
query47	1997	2031	1930	1930
query48	380	421	334	334
query49	715	482	409	409
query50	661	694	404	404
query51	7319	7121	7220	7121
query52	106	111	97	97
query53	237	256	187	187
query54	483	503	423	423
query55	84	78	76	76
query56	255	269	268	268
query57	1265	1227	1160	1160
query58	239	233	227	227
query59	3277	3449	3338	3338
query60	277	263	254	254
query61	110	111	106	106
query62	894	834	762	762
query63	234	192	203	192
query64	3406	1063	654	654
query65	3313	3279	3259	3259
query66	863	402	311	311
query67	16505	16042	15558	15558
query68	9554	767	516	516
query69	501	298	251	251
query70	1276	1114	1060	1060
query71	434	298	248	248
query72	6278	3873	3769	3769
query73	730	755	360	360
query74	10462	9186	8957	8957
query75	4669	3199	2643	2643
query76	5539	1183	782	782
query77	964	380	274	274
query78	9980	10235	9507	9507
query79	2804	906	603	603
query80	708	512	492	492
query81	486	262	231	231
query82	309	149	124	124
query83	195	167	143	143
query84	274	95	70	70
query85	725	362	315	315
query86	348	313	309	309
query87	4440	4447	4453	4447
query88	3407	2234	2210	2210
query89	420	331	290	290
query90	2041	187	184	184
query91	134	133	103	103
query92	67	55	52	52
query93	1394	903	529	529
query94	656	406	288	288
query95	331	262	249	249
query96	490	612	286	286
query97	2771	2841	2729	2729
query98	215	199	193	193
query99	1685	1593	1467	1467
Total cold run time: 297833 ms
Total hot run time: 198296 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7002be21e84b6cb7d7b04c31d44180b833185210, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.42	0.42
query6	1.17	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.51	0.52
query10	0.57	0.57	0.56
query11	0.16	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.84	2.85	2.84
query15	0.92	0.85	0.82
query16	0.37	0.38	0.38
query17	1.07	1.06	1.06
query18	0.23	0.21	0.22
query19	1.95	1.87	2.00
query20	0.01	0.02	0.01
query21	15.36	0.95	0.60
query22	0.76	0.78	0.67
query23	15.30	1.49	0.59
query24	2.77	1.93	1.55
query25	0.23	0.14	0.10
query26	0.18	0.16	0.14
query27	0.06	0.04	0.04
query28	14.37	1.46	1.04
query29	12.57	3.98	3.25
query30	0.26	0.09	0.07
query31	2.82	0.58	0.38
query32	3.24	0.56	0.46
query33	3.11	3.07	3.08
query34	16.67	5.12	4.48
query35	4.49	4.49	4.46
query36	0.62	0.49	0.49
query37	0.10	0.07	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.02	0.03
Total cold run time: 106.38 s
Total hot run time: 32.1 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as draft December 25, 2024 06:33
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review December 25, 2024 07:25
@doris-robot
Copy link

TPC-H: Total hot run time: 33099 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad26051ebd09606b8278f1fe77f9884f2df4e858, data reload: false

------ Round 1 ----------------------------------
q1	17594	6337	6125	6125
q2	2050	309	178	178
q3	10402	1267	760	760
q4	10206	892	462	462
q5	7517	2219	2010	2010
q6	212	184	157	157
q7	914	769	613	613
q8	9247	1418	1228	1228
q9	5345	4994	5047	4994
q10	6806	2296	1871	1871
q11	504	280	263	263
q12	367	365	230	230
q13	17763	3576	2999	2999
q14	243	239	211	211
q15	574	509	498	498
q16	649	633	595	595
q17	590	878	328	328
q18	7317	6549	6477	6477
q19	3039	979	580	580
q20	311	330	189	189
q21	3078	2234	2010	2010
q22	368	351	321	321
Total cold run time: 105096 ms
Total hot run time: 33099 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6374	6234	6290	6234
q2	239	331	233	233
q3	2278	2657	2367	2367
q4	1433	1860	1365	1365
q5	4393	4813	4926	4813
q6	199	176	145	145
q7	2078	1966	1838	1838
q8	2676	2837	2771	2771
q9	7309	7336	7404	7336
q10	3097	3383	2765	2765
q11	589	506	497	497
q12	655	761	631	631
q13	3412	3759	3139	3139
q14	288	311	280	280
q15	589	523	500	500
q16	668	715	641	641
q17	1264	1762	1277	1277
q18	7810	7547	7240	7240
q19	894	1159	1181	1159
q20	2024	2092	1938	1938
q21	5891	5365	5017	5017
q22	654	633	583	583
Total cold run time: 54814 ms
Total hot run time: 52769 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198008 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ad26051ebd09606b8278f1fe77f9884f2df4e858, data reload: false

query1	1312	983	970	970
query2	6493	2407	2375	2375
query3	11080	4901	4719	4719
query4	32955	23959	23521	23521
query5	3698	627	480	480
query6	275	208	193	193
query7	3982	525	308	308
query8	310	253	250	250
query9	9331	2725	2725	2725
query10	446	321	266	266
query11	17912	15373	15283	15283
query12	177	109	115	109
query13	1648	566	412	412
query14	10546	8184	7552	7552
query15	267	213	182	182
query16	7971	612	518	518
query17	1607	771	584	584
query18	2193	415	318	318
query19	206	195	176	176
query20	126	122	106	106
query21	216	129	125	125
query22	4768	4502	4364	4364
query23	34485	33537	33610	33537
query24	6364	2290	2331	2290
query25	504	453	403	403
query26	726	288	155	155
query27	2004	498	345	345
query28	5601	2489	2482	2482
query29	647	582	449	449
query30	209	190	154	154
query31	973	941	832	832
query32	82	63	59	59
query33	485	399	302	302
query34	820	899	536	536
query35	791	881	750	750
query36	1039	1065	990	990
query37	126	103	84	84
query38	4216	4209	4195	4195
query39	1576	1489	1461	1461
query40	236	126	107	107
query41	47	52	45	45
query42	122	105	104	104
query43	528	542	501	501
query44	1376	846	848	846
query45	185	173	177	173
query46	941	1084	681	681
query47	1991	1988	1941	1941
query48	399	419	329	329
query49	721	494	411	411
query50	695	687	400	400
query51	7293	7353	7211	7211
query52	103	108	98	98
query53	240	271	189	189
query54	503	527	429	429
query55	82	84	80	80
query56	259	268	287	268
query57	1262	1236	1186	1186
query58	250	245	234	234
query59	3286	3554	3159	3159
query60	269	274	279	274
query61	115	107	111	107
query62	881	838	758	758
query63	240	199	196	196
query64	3230	1054	717	717
query65	3321	3284	3332	3284
query66	837	412	316	316
query67	16644	15853	15494	15494
query68	9064	769	519	519
query69	483	300	259	259
query70	1176	1171	1150	1150
query71	447	296	256	256
query72	6277	3858	3920	3858
query73	663	765	359	359
query74	10491	9249	9063	9063
query75	4407	3156	2690	2690
query76	4149	1215	799	799
query77	793	377	303	303
query78	10490	10176	9463	9463
query79	5001	877	596	596
query80	755	519	427	427
query81	473	280	231	231
query82	628	151	123	123
query83	195	168	145	145
query84	279	100	77	77
query85	796	369	315	315
query86	368	325	304	304
query87	4546	4432	4540	4432
query88	4832	2236	2230	2230
query89	455	333	288	288
query90	1991	195	188	188
query91	170	134	105	105
query92	64	59	51	51
query93	3051	861	524	524
query94	668	396	283	283
query95	337	259	248	248
query96	490	616	276	276
query97	2730	2831	2695	2695
query98	227	216	201	201
query99	1675	1554	1452	1452
Total cold run time: 302394 ms
Total hot run time: 198008 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ad26051ebd09606b8278f1fe77f9884f2df4e858, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.45	0.41
query6	1.19	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.50
query10	0.55	0.55	0.55
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.61	0.61
query14	2.87	2.74	2.84
query15	0.90	0.83	0.83
query16	0.38	0.38	0.39
query17	1.07	1.08	1.07
query18	0.22	0.20	0.21
query19	1.96	1.87	1.97
query20	0.02	0.01	0.01
query21	15.36	0.92	0.58
query22	0.75	0.78	0.65
query23	15.35	1.39	0.56
query24	2.62	1.30	1.14
query25	0.12	0.23	0.09
query26	0.30	0.16	0.13
query27	0.05	0.06	0.05
query28	13.85	1.53	1.05
query29	12.55	3.94	3.24
query30	0.25	0.08	0.06
query31	2.83	0.59	0.38
query32	3.24	0.54	0.47
query33	3.09	3.12	3.16
query34	16.64	5.11	4.52
query35	4.51	4.55	4.60
query36	0.69	0.49	0.51
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 105.75 s
Total hot run time: 31.71 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32916 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e82bc2518274dc83e8530dffccce245cb9ed9ebe, data reload: false

------ Round 1 ----------------------------------
q1	7539	6045	6077	6045
q2	1854	289	174	174
q3	9902	1237	712	712
q4	6467	856	450	450
q5	1816	2153	2011	2011
q6	207	184	148	148
q7	905	749	611	611
q8	1039	1376	1226	1226
q9	5017	5007	4959	4959
q10	2261	2329	1861	1861
q11	400	272	266	266
q12	332	355	228	228
q13	3304	3579	2981	2981
q14	227	227	225	225
q15	564	522	505	505
q16	634	618	593	593
q17	562	868	333	333
q18	7019	6496	6566	6496
q19	611	976	564	564
q20	319	317	200	200
q21	2904	2243	2017	2017
q22	371	338	311	311
Total cold run time: 54254 ms
Total hot run time: 32916 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6249	6269	6245	6245
q2	240	324	230	230
q3	2256	2592	2331	2331
q4	1405	1784	1357	1357
q5	4405	4765	4721	4721
q6	182	172	141	141
q7	1942	1865	1752	1752
q8	2540	2672	2638	2638
q9	7011	6893	6940	6893
q10	2954	3246	2729	2729
q11	561	515	504	504
q12	632	740	575	575
q13	3222	3659	3000	3000
q14	273	298	284	284
q15	581	500	496	496
q16	642	695	645	645
q17	1193	1700	1228	1228
q18	7467	7261	6955	6955
q19	787	1067	1108	1067
q20	1971	2018	1829	1829
q21	5464	5158	4957	4957
q22	644	592	563	563
Total cold run time: 52621 ms
Total hot run time: 51140 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190527 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e82bc2518274dc83e8530dffccce245cb9ed9ebe, data reload: false

query1	975	387	378	378
query2	6522	2379	2219	2219
query3	6705	210	206	206
query4	33840	23767	23508	23508
query5	4325	616	471	471
query6	276	205	185	185
query7	4624	494	299	299
query8	304	267	235	235
query9	9582	2757	2735	2735
query10	477	310	266	266
query11	18168	15800	15148	15148
query12	165	105	103	103
query13	1676	531	431	431
query14	12370	7778	6876	6876
query15	248	191	191	191
query16	8179	582	437	437
query17	1585	750	600	600
query18	2103	407	340	340
query19	226	187	147	147
query20	112	107	112	107
query21	213	124	105	105
query22	4375	4441	4373	4373
query23	34599	33435	33401	33401
query24	6026	2270	2286	2270
query25	479	437	373	373
query26	1202	287	151	151
query27	1989	462	335	335
query28	5357	2440	2431	2431
query29	696	540	435	435
query30	228	183	151	151
query31	1016	884	814	814
query32	73	62	59	59
query33	531	371	310	310
query34	747	837	506	506
query35	786	838	728	728
query36	1019	1022	957	957
query37	116	97	72	72
query38	4235	4228	4109	4109
query39	1515	1415	1426	1415
query40	206	116	100	100
query41	47	44	48	44
query42	123	105	105	105
query43	503	532	477	477
query44	1283	802	799	799
query45	179	167	166	166
query46	869	1053	640	640
query47	1931	1948	1892	1892
query48	380	394	334	334
query49	785	485	376	376
query50	621	652	385	385
query51	7036	7084	6971	6971
query52	102	100	92	92
query53	222	252	196	196
query54	481	503	398	398
query55	84	77	77	77
query56	256	264	238	238
query57	1191	1188	1135	1135
query58	236	229	227	227
query59	2918	3125	3148	3125
query60	275	252	243	243
query61	106	109	106	106
query62	961	785	743	743
query63	224	195	198	195
query64	4364	970	632	632
query65	3297	3240	3216	3216
query66	1049	416	319	319
query67	15876	15935	15594	15594
query68	8742	749	506	506
query69	451	297	260	260
query70	1230	1117	1066	1066
query71	424	288	262	262
query72	6106	3864	3831	3831
query73	668	760	368	368
query74	10539	9350	9272	9272
query75	4524	3154	2629	2629
query76	4523	1207	789	789
query77	743	354	374	354
query78	10252	10259	9451	9451
query79	3736	954	577	577
query80	717	514	425	425
query81	483	278	230	230
query82	638	164	124	124
query83	204	163	146	146
query84	283	91	75	75
query85	782	365	299	299
query86	366	332	298	298
query87	4452	4587	4299	4299
query88	4415	2224	2223	2223
query89	405	333	310	310
query90	1904	187	178	178
query91	133	128	109	109
query92	70	55	54	54
query93	1749	870	530	530
query94	660	397	286	286
query95	325	264	244	244
query96	493	594	279	279
query97	2793	2828	2704	2704
query98	231	204	194	194
query99	1644	1529	1453	1453
Total cold run time: 297954 ms
Total hot run time: 190527 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e82bc2518274dc83e8530dffccce245cb9ed9ebe, data reload: false

query1	0.04	0.03	0.05
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.60	0.11	0.10
query5	0.42	0.41	0.42
query6	1.14	0.68	0.66
query7	0.02	0.02	0.02
query8	0.04	0.02	0.04
query9	0.59	0.51	0.51
query10	0.54	0.58	0.57
query11	0.15	0.10	0.10
query12	0.14	0.12	0.12
query13	0.61	0.62	0.60
query14	2.83	2.86	2.78
query15	0.90	0.83	0.83
query16	0.40	0.37	0.38
query17	1.05	1.08	1.06
query18	0.22	0.21	0.22
query19	1.87	1.80	1.98
query20	0.02	0.01	0.01
query21	15.35	0.93	0.59
query22	0.76	0.79	0.76
query23	15.19	1.41	0.57
query24	2.93	1.20	1.18
query25	0.20	0.09	0.09
query26	0.34	0.14	0.14
query27	0.04	0.05	0.05
query28	13.44	1.55	1.05
query29	12.60	3.99	3.31
query30	0.25	0.09	0.07
query31	2.82	0.59	0.39
query32	3.23	0.55	0.46
query33	3.08	3.04	3.11
query34	16.89	5.10	4.43
query35	4.48	4.48	4.46
query36	0.63	0.48	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.6 s
Total hot run time: 31.67 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@yiguolei yiguolei yiguolei left review comments

@wangbo wangbo wangbo left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zhiqiang-hhhh @doris-robot @yiguolei @wangbo